Skip to content
This repository has been archived by the owner on Jun 23, 2022. It is now read-only.

fix: crash when huge write #1099

Merged
merged 4 commits into from
Apr 28, 2022
Merged

fix: crash when huge write #1099

merged 4 commits into from
Apr 28, 2022

Conversation

ZhongChaoqiang
Copy link
Contributor

Related-Issue

apache/incubator-pegasus#903

Change

when buffer of slog is more 2GB, then flush to slog file, it maybe crash.

foreverneverer
foreverneverer previously approved these changes Apr 26, 2022
View reviewed changes
acelyc111
acelyc111 reviewed Apr 26, 2022
View reviewed changes
src/aio/disk_engine.cpp Outdated Show resolved Hide resolved
@acelyc111
Copy link
Member

@ZhongChaoqiang The CI failed at https://github.com/XiaoMi/rdsn/runs/6189338284?check_suite_focus=true#step:8:1807, plz check it.

@ZhongChaoqiang
Copy link
Contributor Author

ZhongChaoqiang commented Apr 27, 2022 via email •
edited by acelyc111
Loading

@acelyc111
Copy link
Member

acelyc111 commented Apr 27, 2022 •
edited
Loading

@ZhongChaoqiang
See details here, it tells you what the problem is.
The variable in src/aio/disk_engine.cpp L269 is uint32_t sz;, but in src/aio/disk_engine.cpp L56, we want to dereference it as an uint64_t (uint64_t &sz = *(uint64_t *)plength;). So stack-buffer-overflow error occurs, and may cause following coredump.

@ZhongChaoqiang
Copy link
Contributor Author

@ZhongChaoqiang See details here, it tells you what the problem is. The variable in src/aio/disk_engine.cpp L269 is uint32_t sz;, but in src/aio/disk_engine.cpp L56, we want to dereference it as an uint64_t (uint64_t &sz = *(uint64_t *)plength;). So stack-buffer-overflow error occurs, and may cause following coredump.

Thanks very much

@acelyc111
Copy link
Member

Thanks for your contribution!
Could you describe how do you troubleshooting the bug you found when write with huge thoughput, and create a specify issue?

@xihong08
Copy link

xihong08 commented Apr 28, 2022 •
edited
Loading

Thanks for your contribution! Could you describe how do you troubleshooting the bug you found when write with huge thoughput, and create a specify issue?
@acelyc111 . i can describe detailed case. this maybe happened on the server side in busying write status，and crash happened in aio_task::collapse ::memcpy(dest, b.buffer, b.size); it resulting by memory overwriting . and root cause is integer overflow，coz aio_context's buffer_size is 4 bytes, in dsn::file::write_vector file_io.cpp L128 , the aio_context's buffer_size may be overflow when to be written data size more 4GB, so crash it later.

@acelyc111 acelyc111 merged commit 3f0c632 into XiaoMi:master Apr 28, 2022
@foreverneverer foreverneverer mentioned this pull request Jul 15, 2022
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@foreverneverer foreverneverer foreverneverer approved these changes

@acelyc111 acelyc111 acelyc111 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ZhongChaoqiang @acelyc111 @xihong08 @foreverneverer